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Abstract 

A dynamic system, which is used in the neural network theory, Ising spin glasses 
and factor analysis, has been investigated. The properties of the connection matrix, 
which guarantee the coincidence of the set of the fixed points of the dynamic system 
with the set of local minima of the energy functional, have been determined. The 
influence of the connection matrix diagonal elements on the structure of the fixed 
points set has been investigated. 

1 Introduction 

We define a neural network as a dynamic system of n spin variables (spins) which can 
take one of two values: 

a l = {±l}, z = l,2,...,n. (1) 
The spins are connected by a symmetric connection matrix J = (Jij) : 

J%j Jji j i i j 1 , 2 , . . . , 71 . 

The local potential 

n 

= £-W*) (2) 

3=1 

with which the network acts on spin i, determines solely the value of spin i at time t+1: 

A + '~ \ ~(Ti(t), if hitycTiit) < [6) 

The state of the network as a whole is described by a configuration vector a, whose 
coordinates are given by Eq.(l). In what follows, Greek letters will be used to designate 
configuration vectors. 

We want to investigate the set of the so called fixed points of the network, i.e. such 
states a*, that for all coordinates a* have: 

a*hi>0, i = l,2,...,n. (4) 
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Besides the neural network theory, the mathematical model (l)-(3) is also used in the 
factor alalysis [[TJ and in the Ising spin glass theory . The neural network theory makes 
use of a physical concept of the energy of the state a, which is defined as 

-i n 1 n 

E{a) = VVi/ii = V JijCTiCTj. (5) 

It is very important, both from the physical point of view and the ability of the 
network to have content-addressable memory that the energy of the state would be 
a decreasing function on every step of the network evolution. And, moreover, the fixed 
points must be the local minima of the energy functional (5). 

In the second section, we obtain the conditions under which a connection matrix J 
guarantees the fulfillment of the above mentioned requirements. It has been found that 
the Hebb connection matrix as well as the connection matrices which are used in physical 
problems possess the necessary property. But this is not the case for the projection 
matrix 0. As a result, a network with such a connection matrix has a set of fixed points 
which is wider than the set of the local minima of the energy functional (5). In the 
third section, we show how the situation for a network with a projection matrix can be 
improved. 

Notations. In what follows, a network with a connection matrix J is called a J- 
network. We denote by FP(J) the set of all fixed points of the J-network. A configuration 
vector which is a fixed point of a network will have a superscript "*": 

a* G FP{J). 

We denote by LM(J) the set of the local minima of the energy functional (5). To 
examine the local minima of the functional (5), we introduce a topology on the set of 
configuration vectors: the set of n configuration vectors a"> which are the nearest to the 
vector a in the sense of the Hamming distance will be called a vicinity of the state a: 

a® = (o-i, er 2 , . . . , -<j h ...,a n ), I = 1, 2, . . . , n. 

In other words, the state a® from the vicinity of the state a differs from the latter by the 
opposite value of the /th spin only, 

a G LM(J) <£> E{3) < E(a {l) ), I = 1, 2, . . . , n. 

Finally, a matrix with zero diagonal elements will be marked by the superscript " 0" : 

J(°)^J.. = 0, z = l,2, ...,n. 



2 On the role of the diagonal elements 

Theorem 1 

1. The set of the local minima of the energy functional (5) does not depend on the 
value of the diagonal elements of the connection matrix: 

LM(J) = LM(J + A), 
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where 

A = diag{a u , a 22 , ■ ■ ■ ,a nn } (6) 

and all the elements arbitrary real numbers. 

2. For a connection matrix with zero diagonal elements, the set of the fixed points 
coincides with the set of the local minima of the energy functional 

FP(J (0) ) = LM(J (0) ). (7) 

3. Let all the elements an of the diagonal matrix (6) be positive, then 

FP(J (0) + A) D FP(J (0) ) D FP(J (0) - A). (8) 

The proof of Theorem 1: Let's write the energy of the state 3, extracting the contri- 
bution of the /th spin. Up to the positive factor we obtain: 

E{3) oc - JijViVj + Ju - Verify. (9) 

The state 3 will be the local minimum of the energy functional if and only if the system 
of the inequalities (10) is fulfilled for all states 3® from the vicinity of the state 3: 

E(o®) - E{3) oc dh - J u = a, £ <Vi > 0, I = 1, 2, . . . , n. (10) 

It is evident that the inequalities (10) do not depend on the values of the diagonal elements. 
The conditions of their fulfillment are defined by the off-diagonal part of the matrix J. 
By this the first item of the Theorem is proved. Moreover, it follows from the inequalities 
(10) that for nonnegative Ju any local minimum of the energy functional is also a fixed 
point of the network: 

LM(J) C FP(J) when J u > 0, I = 1,2, . . . ,n. (11) 

Then, let a state 3* be a fixed point of the network: a* hi > 0, I — 1,2, ... ,n. With 
the help of Eq. (9) we obtain: 



E(3 {1) ) - E{3*) oc ^ h - J u { 



{ > 0, if J u = 
? 0, if J u >0 
{ > 0, if J u < 



In other words, for nonpositive Ju any fixed point of the network is a local minimum of 
the energy functional: 

LM(J) D FP(J) when Ju < 0, / = l,2,...,n. (12) 

Combined with the proved first item of the Theorem, Eqs. (11) and (12) justify the 
correctness of Eqs. (7) and (8). Thus, the proof of the Theorem is finished. 

In fact, to some extent the Theorem 1 permits regulating the set of the fixed points 
of the network. Let's explain this statement. If the matrix is transformed by the 
diagonal matrix A, J (A) = J^ + A, then in accordance with the Theorem 1, the set of 
the local minima of the energy functional is not changed. But this transformation affects 
the set of the fixed points of the J(A)-network. Indeed, if all the matrix elements an are 
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positive, the set of the J(A)-network fixed points extends as compared with FP(J^). 
The last is true due to the appearance of the new fixed points which are not the local 
minima. If, on the contrary, all the matrix elements an are negative, the set of the fixed 
points of the J(A)-network narrows as compared with FP(j(°>): some states, remaining, 
as they were, the local minima of the energy functional, cease to be the fixed points. 

The last statement allows to suggest a simple method for the elimination of the un- 
necessary fixed points of the network. Let's formulate it in the form of a theorem. 

Theorem 2 

Let the fixed points of a j( )-network be numbered in such a way that 

E{3® < £(<r (2) <...< E{3^ < E(a^ <... 

(to simplify the writing, here we omit the superscript "*" in the notations of the fixed 
points). Let A be a diagonal matrix whose elements are defined by the equalities 

a„ — min af'hi(a^), z = l,2, ...n, (13) 

l=l,k 

where hi(a) is the potential (2) which acts on the ith spin in the -network. Then 

FP(J(°) -A) = ^,...,^1}. 

The proof of Theorem 2: Since all an from Eq. (13) are positive, the set of the 
(j(°) — j4)-network fixed points, due to Theorem 1, can be only narrower in comparison 
with the set -network fixed points. And from the definition (4) of a fixed point, it 
follows that the state a® will be a ( — A)-network fixed point if and only if the system 
of the inequalities 

aVhiiff®) - a u > 0, i = l,2,...,n (14) 

is fulfilled. When the definition (13) is taken into account, it is evident that for any fixed 
point a® with the I < k the system of the inequalities (14) is fulfilled. Consequently, 
the first k states of the a"' are the (J^ — A)-network fixed points. On the other hand, 
by proceeding from Eq.(5) for the energy and taking into account that the -network 
fixed points are strictly ordered with respect to the energy increase, it is easy to see that 
at least for one of the coordinates of the state with I > k, the inequalities (14) are 
not fulfilled. Consequently, the states a® with I > k are not the ( — A)-network fixed 
points. 

Remark. From Eq.(9) it can be easily shown that even under the sequential dy- 
namics the evolution of a network with a connection matrix whose diagonal elements are 
negative can be accompanied by the energy increase. As a result, even under the sequen- 
tial dynamics limit cycles can be formed for such a network! From this point of view, 
the connection matrices with negative diagonal elements are absolutely nonphysical. But 
Theorem 2 gives a simple and effective method to eliminate high energy fixed points. In 
some cases this method can be very useful. In particular, it is well-known |], , that for 
a network with the projection connection matrix the energies of the spurious fixed points 
are larger then the energies of the memorized patterns. Consequently all such spurious 
fixed points can be easily eliminated with the help of Theorem 2. 
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3 Projection connection matrix 



1). Let 

F> = $>,$,...,$), l = l,2,...,p, (15) 

be p preassigned configuration vectors which we would like to have as a network fixed 
points (such vectors are usually called the memorized patterns). It is known ||, that 
it can be easily done if a matrix P of orthogonal projection onto the linear subspace 
A < |*(i) |*( 2 ) |*(p)>> spanned by the p memorized patterns is taken as a connection matrix. 
The matrix P is symmetric and nilpotent one: P T = P, P 2 = P. Besides, by definition 
P£^ = I = 1,2, ... ,p. Consequently, the vectors £^ are not only the fixed points 

of the P- network, but provide the global minimum of the energy functional (5): 

= -i£L-l5-J = -1, I = 1,2,..., p. 
n 

But, it is known from experience, that, as a rule, the P-network has additional fixed 
points which are called spurious fixed points. Their number is much larger that for the 
network with Hebb's connection matrix. And the worst is that not all the P-network 
fixed points are the local minima of the energy functional ||. Theorem 1 helps to clarify 
the situation. 

2). For simplicity we assume that p memorized patterns £® are linearly independent 
vectors. We introduce a rectangular (p x n)- matrix 5 whose rows are memorized patterns 
|t0- 

$ ■■■ e\ 

A 2 ) A 2 ) <r(2) 

SI "52 • • • Qn ^6) 

Then the matrix of the orthogonal projection onto the linear subspace A^i) g?) |tp)> is 

P = YE, (17) 

where Y is the (n x p)-matrix that is pseudoinverse of the matrix H. Apropos of the 
construction of pseudoinverse matrices, see || |J. We only want to mention, that the 
columns of the matrix Y are n-dimensional vectors such that ) = hi'i where 

5u> is the Kronecker symbol; are also the linearly independent vectors. 

The diagonal elements of the matrix P are positive. Indeed, they are equal to the 
squares of the projections onto the subspace A < |- {1) |tri> °f the ^-dimensional Carte- 
sian unit vectors = (0, . . . , 1, . . . , 0): 



Pu = (Pe« e«) =|| Pe« \\ 2 = dl i = 1,2, 



,n. 



The values df must be nonzero, otherwise the vectors Pe® = Sf=i ^t'i which are the 
sums of the linearly independent vectors, would be zero. Then, according to Theorem 1, 
the set of the P-network fixed points will really be wider than the set of the local minima 
of the energy functional. 

If instead the P-network we consider the p( )-network, 

p(°) = p - D, where D = diag{d\, <&..., d 2 n } (19) 
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only the local minima of the energy functional will be the fixed points of the P( )-network. 
Since the memorized patterns provide the global minimum of the energy functional, they 
will necessarily be the P^ ) -network fixed points. In addition, the p(°) -network has no 
nonphysical fixed points, i.e. those which are not the local minima of the energy functional. 

Let's show that the set of the P^-network fixed points has a structure which resembles 
the structure of the set of the fixed points for Hopfield's model ||. 

Theorem 3 

All n neurons can be enumerated in such a way that all the P^-network fixed points 
will be of a "piecewise constant" kind: 



a 



(£l, £1, . . . , £1, £2, £2, 



1 £2, • • • 5 &V 1 &V 1 • • • 1 &v ) 1 



(20) 



n-2 



where £j = {±1}. The number v, the composition and the dimension rij of constant sign 
intervals are defined uniquely by the memorized patterns matrix H (16). 

The proof of Theorem 3. For simplicity we assume that the number of the neurons n 
is much larger than the number of the memorized patterns p. For definiteness, we assume 
that n > 2 P . Then it is evident that not all n p-dimensional column-vectors 



( d 1} \ 



V £?° J 



1,2, 



n 



(21) 



of the matrixS will be different. Some of them will be repeated []. Let's assume that n 
column-vectors & break up into v groups. Each group consists of some identical vectors: 
rii identical vectors belong to the first group; n 2 identical vectors belong to the second 
group and so on. And n\ + n 2 + . . . + n v = n. Without loss of generality it may be 
assumed that the neurons of the network can be numbered in such a way that the matrix 
H will have a form: 



t( 2 ) £\* 
SI • • • SI 

Vsf } ••• d p) & p) ... &> ••• ti p) ••• e ] ) 

that when the projection matrix P acts onto the configuration vector 



r(2) 



S2 
d(2) 
S2 



S2 

t(2) 
S2 



St! 

Sj; 



Si; 

f(2) 
Sj; 



:22) 



It was shown in 
a the result is 



Pa 



v 

1=1 



V) 



where m"'(a) depends only on the state a. If a* is a fixed point of the P^ -network, its 
coordinates a* must satisfy the system of equations: 



sign{Pa* - Da*) = sign [Y,m {l) {a*)i 



(i) 



•2 * 



di<r. 



1,2, 



n. 



(23) 



^^The p-dimentional column- vectors ^ (21) are labelled with subscripts in contrast to the n-dimensional 
memorized patterns (15), which are labelled with superscripts. 
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With respect to the form of the matrix E (see Eq.(22) ), it is easy to understand that, 
for example, for the first n\ values of the subscript % all the sums J2f=i m^(a*)^ are 
the same: 2f =1 w^(<7*)£^ = M, i = 1,2, From here it immediately follows, 

that the first n\ coordinates of the fixed point a* must be equal. Indeed, let's contrarily 
assume that, for example, a\ = 1 and a\ = —1. Then from Eqs. (23) it follows that two 
inequalities 

M > d\ 
-M > d\ 

have to be fulfilled simultaneously. But this is impossible. 

Consequently, the first n\ coordinates of the P^-network fixed point are equal to one 
another. Similarly, it can be proved that the next n 2 coordinates of the fixed point are 
equal to one another too, and so on up to the equality among the last n v coordinates. 
The Theorem is proved. 

3). Thus, the P^^-network possesses some attractive properties. Namely, all the 
memorized patterns will necessarily be its fixed points. Moreover, all the fixed points 
are, firstly, the local minima of the energy functional; secondly, they are of "piecewise 
constant" kind given by Eq. (20). It is known j|, |j that the last two of the above 
mentioned properties are also typical for the fixed points of the Hopfield model, that is, 
for a network with the Hebb connection matrix. In our following discussions we will have 
in mind either the P^-network, or the Hopfield model. 

While Eq.(20) is only the necessary condition for a configuration to be a fixed point 
of a network, it restricts sufficiently the circle of the "applicants". Let's provide some 
estimates. 

If v is the number of different column-vectors of the matrix H and q is the number of 
the fixed points of the related network, it is clear that q < 2 V . Next, when the number 
of the memorized patterns p is given, a natural estimate for v is: v < 2 P . But, as it was 
shown in ||, actually we have: 

q<2 2P ~\ (24) 

Eq. (24) restricts the number of the fixed points from above. It is easy to verify that 
when p = 2, the inequality (24) transforms into the equality, and q just equals 4. But 
even for p = 3 the right-hand side of Eq.(24) is 16, and in Q it was shown that for p = 3 
the maximum possible number of the network fixed points is 14. This result coincides 
with the one of 0j. Other estimates for the fixed points number can be found in M, |9], |I(J 
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